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ABSTRACT 



The New York Collaborative for Excellence in Teacher 
Preparation (NYCETP) is a project jointly undertaken by five college campuses 
of the City University of New York and New York University to enhance teacher 
education and to design and develop mathematics and science content and 
education courses for students preparing to be teachers. Over the first 3 
years of the collaborative, the faculty development activities that involved 
cross-campus and cross -discipline collaboration occurred in large 
conferences, workshops, curriculum working groups, and through case studies. 
Formative evaluation has focused on facilitating faculty development, 
providing baseline data on courses, documenting change within the 
Collaborative courses, developing peer reviews of course documents, and to a 
lesser degree, assisting faculty in end-of-year course evaluations. Case 
studies of the Collaborative courses have been an important part of the 
ongoing evaluation, and an outline has been developed to guide faculty 
members in writing case studies. Evaluation of the CETP is important for both 
funders and evaluators. Both funders and evaluators are interested in careful 
review of project proposals and early warning systems for potential problems. 
Evaluators may have a more consultative function, but funders, who have 
responsibility to a broader constituency, may need to serve more as judges. 
Four appendixes contain a self-study guide, a related glossary, a campus peer 
review summary table, and the case study outline. (Contains 1 table and 26 
references.) (SLD) 
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The New York Collaborative for Excellence in Teacher Preparation: 

A Case Study of Context and Evaluation 
Introduction 

Description of NYCETP 

Description of NYCETP context. The New York Collaborative for Excellence 
in Teacher Preparation (NYCETP or "the Collaborative") is a project jointly 
undertaken by five college campuses of the City University of New York (CUNY) 
and New York University (NYU). The project is funded by the National Science 
Foundation (NSF), with additional support from the participating institutions. It is 
presently in its fourth year of a five-year grant. The main goals of the Collaborative 
to date include: (1) collaboration within and between the five campuses of CUNY 
and NYU; (2) faculty development emphasizing curriculum and teaching standards 
such as the National Council of Teachers of Mathematics (NCTM, 1989) and the 
National Research Council (NRC, 1996) Standards documents; and (3) the design 
and development of mathematics and science content and education courses for 
students preparing to become teachers. Although there are other important goals 
(e.g., recruitment of math and science teachers and increasing numbers of 
underrepresented populations within the teaching profession), these specific 
objectives form the basis for the Liberal Arts and Science course reform and 
programmatic change that are primary to the NSF CETPs' purpose. 

The Collaborative includes colleges from a large public urban institution and a 
large private urban university. The five CUNY campuses are situated within four of 
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the five boroughs of New York City (Brooklyn College in Brooklyn, Lehman College 
in the Bronx, City College of New York and Hunter College in Manhattan, and the 
College of Staten Island on Staten Island), and NYU is located in Manhattan. 
Although each of these colleges is situated within New York City, they are located 
within vastly different areas of the city. Thus, the student bodies of each are quite 
different. Two of the CUNY campuses, Lehman College and City College are 
located in predominantly multiethnic areas, and Lehman College students are 
mostly second language learners. On the first New York State teacher licensure 
exam, the Liberal Arts and Sciences Test (LAST - Basic Skills; 1997), both of these 
colleges had a high percentage of failures. Brooklyn College and Hunter College 
also draw from a largely multiracial population; however, the students from these 
campuses performed much better on this indicator with over 80% of the education 
students passing the LAST on its first administration. The College of Staten Island 
is not an ethnically diverse campus. The college had a high rate of success on the 
LAST. 

The student populations of CUNY and NYU vary considerably. While CUNY 
campuses draw almost solely from the New York area, NYU draws its student 
population from both the New York metropolitan area as well as nationwide. NYU 
is a selective private institution, and virtually 100% of its education students 
passed the LAST. The teacher education program at each of the participating 
universities is focused mainly on urban education. 

The project director is a tenured faculty member on one of the CUNY 
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campuses and is one of six co-principal investigators (Pis). During the second year 
of the Collaborative, the director also served as the Dean of Education on her 
campus. Of the other Pis, one is a Dean of Education and the remainder are 
tenured faculty. Functionally, NYU, Hunter College and the College of Staten Island 
have two faculty members serving as co-PIs, one in the Arts and Sciences and one 
in Education. These individuals, all of whom manage their responsibilities on 
summer salary time and/or course release time, constitute the Executive Board, 
which oversees the activities of NYCETP. The project director holds primary fiscal 
and governance responsibility, supported by a full-time project coordinator and an 
administrative assistant. The co-PIs are responsible for the activities and 
involvement of the faculty on their campus, campus strategic planning, campus 
annual report data, coordinating recruitment, dissemination, and Collaborative-wide 
activities such as the Teaching Scholars program. The campus Pis are supported by 
one or two faculty members designated as campus coordinators. 

Faculty development activities Year One. During the first year (1 995-1 996) 
of the Collaborative there were four faculty workshops and two larger conferences 
held at the different Collaborative institutions. These activities represented 
situations that brought faculty from various campuses together in a formal 
workshop or conference setting. The faculty workshops provided opportunities for 
faculty on each of the campuses to present activities taking place on their campus 
and thus served as a means for disseminating information related to project goals 
that were already underway at the time. Each of these workshops was attended by 
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between 20 and 30 faculty members from the six campuses. The two conferences 
held during the first year provided contexts for college faculty and administration, 
as well as public school teachers and administration, to learn about the integration 
of methods of teaching and content, in the first case, and the integration of 
technology and instruction, in the second case. Forty-six (out of 91) participants 
who attended the first conference completed a feedback survey, including 19 
college faculty, four District Coordinators of Mathematics or Science, and 20 New 
York Public School teachers. There were 62 participants at the second conference 
including 34 college faculty, nine public school administrators, and 13 public school 
teachers. 

In addition, during the spring and summer of the first year, the evaluators 
instituted faculty case studies. Faculty members from the various campuses were 
identified to study courses on other campuses that were proposed for revision the 
following year. These case studies were an attempt to directly facilitate the goals 
of the Collaborative, and to provide baseline data. Faculty members were identified 
to write the case studies and to be "case studied" with the goal of fostering 
interactions between individuals teaching similar courses at different institutions. 
This served both the goals of faculty development and evaluation. 

Faculty development activities Year Two. During year two (1996-1997), the 
Collaborative members engaged in two faculty workshops, one conference, and 
began to meet as Curriculum Working Groups within specialty areas. The 
workshops focused on practical classroom topics such as the use of technology in 
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education and assessment issues, and were attended by less than 20 college 
faculty each. This attendance number was smaller than year one. The Curriculum 
(faculty) Working Groups were for elementary mathematics, elementary science, 
and secondary mathematics and science. They were formed to discuss course 
revisions, to provide feedback on course documentation, and, for the secondary 
group, to discuss issues of recruitment and retention. The faculty attendance 
ranged from a high of 29 (including Pis) in February 1 997 to lows of 1 5 in April 
and 1 3 in May, 1 997. 

The large conference for the year was held at an informal science institution, 
the New York Hall of Science, "Mathematics and Science for All: How Informal 
Science-Rich Institutions Can Contribute to National Reform Efforts in Mathematics 
and Science." It was attended by 161 educators, 99 participants identified their 
affiliations, which included 47 public school teachers, seven public school 
administrators, 27 college faculty, four college students, and five museum 
personnel. 

Four revised courses were "Case Studied" as in year one, and a peer review 
process for evaluating and providing feedback to course developers was used. This 
peer review was developed by the evaluators to function as an indicator of the 
quality of the revised courses with respect to the Collaborative goals. 

Faculty development activities Year Three. The faculty development 
activities during year three (1997-1998) followed the pattern of the previous two 
years — one conference. Curriculum Working Group meetings that continued the 
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peer review of course documentation, and three evaluation case studies. The 
annual conference was held at the American Museum of Natural History and 
focused on inquiry as teaching. The 42 participants included 22 public school 
teachers and 14 college faculty. 

In summary, over the first three years, the faculty development activities 
that involved the cross-campus and cross-discipline (i.e., between Education and 
Liberal Arts and Sciences faculty) collaboration occurred in the large conferences, 
workshops, curriculum working groups, and through the case studies. The 
sustained Collaborative interaction occurred for the Pis, who met frequently (i.e., at 
least monthly) in years one and two, and somewhat less frequently in year three. 
The Pis also met with an internal faculty advisory group four to six times per year, 
and an external advisory group one to two times per year. Thus, over all the project 
activities there is a stable core of approximately 20 faculty members at meetings 
and little cross-campus faculty interaction outside of these scheduled meetings. 

Evaluators for NYCETP. During the first three years of the Collaborative, 
there were two internal evaluators and one external evaluation group on the 
NYCETP project. The internal evaluators are a faculty member and a doctoral 
student at the Graduate School and University Center, CUNY. The internal 
evaluation staff is funded by a $40,000 (direct costs) subcontract for the NYCETP 
formative, internal evaluation to the Center for Advanced Study in Education, a 
research and development center affiliated with the educational psychology 
doctoral program at the Graduate School. The NYCETP project budget also includes 
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annual funds of $25,000 for the external evaluation team, a function that has been 
filled by the Evaluation and Training Institute (ETI) for years two and three of the 
project. 

Overall, the formative evaluator's role and relationship to the Collaborative is 
as a consultant providing feedback, documentation, and development of evaluation 
activities that directly support specific Collaborative goals, as with the Case 
Studies and the peer review process of courses. This has involved frequent 
interactions with the project coordinator (in addition to the PI) who consults on 
agendas for meetings, reactions to conferences, and strategies for focusing 
activities on the project goals. Thus, the evaluators' role has been atypical for a 
formative evaluator in reaching out and working with staff other than the PI. This 
atypical role has been taken on since the PI, with many outside responsibilities, 
lacks sufficient time to deal with details of running the project. 

Typically, the evaluators attended the PI, Executive Board, and Internal 
Advisory Board meetings, workshops, conferences, curriculum group meetings, 
etc., and provided immediate feedback as appropriate. These comments focused on 
discrepancies between Collaborative goals and implementation of particular 
activities, and, after year one, the goals as emphasized by NSF and the NSF 
National Visiting Committee. 

Evaluation Models/Theory Used 

The governance structure of NYCETP and the diversity among and between 
the campuses and their faculties played an important role in developing evaluation 
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activities. The early (and continuing) evaluation focused on formative feedback and 
documentation of faculty development activities. However, by the end of Year One, 
the primary evaluation activities were explicitly designed to encourage and support 
the goals of collaboration and course revision. Following Weiss (1997) and Lipsey 
(1993, 1997), we focused on faculty and course change for theory-based 
evaluation activities and involved faculty in carrying out the activities to support 
use of the evaluation (Patton, 1997; Weiss, 1998). Although there were other 
evaluation activities such as the examination of student attitudes following a 
reformed course and surveys of student views of a specific course, the crucial 
evaluation strategies with respect to the fidelity of course reform were the faculty 
case studies and peer reviews for new, revised or to-be-revised courses. 

Within non-experimental applied research contexts and particularly when 
educational reform is the object of evaluation, theory plays an important role in 
strengthening evaluations (Cook & Campbell, 1979; Lipsey, 1993). Cook and 
Campbell explicitly identify the issues of the construct validity of causes 
(treatments) as well as effects. The idea of construct validity of the treatment, the 
"black box" or treatment theory characterized by Lipsey, is that the causal analysis 
is "... strengthened by an explicit theory about the nature and details of the change 
mechanism through which the cause of interest is expected to produce the 
effect(s) of interest" (Lipsey, 1993, p. 6). 

Current educational reform derives from changing perspectives on teaching 
and learning. The "black box" in the Collaborative's reform efforts is the classroom. 
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The NCTM (1989, 1991, 1995) and NRC (1996) Standards documents are based 
on theories of knowledge construction and instruction that can be broadly 
characterized as developmental and apprenticeship in their orientation (Farnham- 
Diggory, 1994) or constructivist, emergent and sociocultural (Cobb & Yackel, 

1996). The roles of teacher and student, classroom interactions and tasks, and the 
nature of student outcomes are the focus of reform. 

Similarly, the role of theories or models of evaluation are also under revision. 
For example, O'Sullivan (1995) examines the emerging roles of evaluation in 
science education reform, Frechtling (1995; Frechtling & Sharp, 1997) considers 
the strategies for non-traditional program evaluation, and others have archived 
reform projects through the use of case studies (Stake, et al., 1993; Trumbull, 
1993a, 1993b). From these perspectives, evaluation efforts can serve to foster and 
shape reform efforts. 

Thus, the formative evaluation practices of the NYCETP have been focused 
on facilitating faculty development, providing baseline data on courses, 
documenting change within the Collaborative courses, developing peer reviews of 
course documents, and, to a much lesser degree, assisting interested faculty in end 
of year course evaluations. Formative feedback has emphasized course 
development resources and substantive products. The faculty case studies were, 
and are, an effort to directly facilitate the goals of the Collaborative. That is, they 
were carried out to facilitate the attainment of the first two goals, fostering 
collaboration between faculty on different campuses and facilitating faculty 
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development. They were also developed to stimulate the Collaborative's efforts 
toward the third goal, development and/or revision of curriculum and classroom 
practice. Faculty members were identified to participate in the case studies with 
the goal of fostering interactions between faculty in the Liberal Arts and Sciences 
and Education. In conjunction with the case studies, NYCETP faculty members and 
evaluators began to "peer review" course documentation, an activity also intended 
to serve the goals of faculty development, faculty collaboration, and, to a limited 
degree, facilitate the development of course reform. 

Evaluation Outcomes 

The evaluation practices that have directly impacted course reform and 
which may help us to gauge the fidelity of course reform and the role of context 
include: (1) workshop and conference attendance and feedback; (2) product 
specifications for NYCETP course dissemination; (3) peer reviews of reformed 
courses; and (4) case studies of reformed courses. The latter two provide an in- 
depth view of the reformed courses that can be examined for the fidelity with 
which these courses adhere to the goals of the Collaborative and the national 
reform movements within mathematics and science education. 

Workshop and conference feedback. For the first three years, the main 
NYCETP activities were workshops, conferences, and curriculum working groups. 
These activities were the Collaborative’s faculty development program, providing 
forums for display, examination, and discussion related to particular courses and 
teaching techniques. They constituted an attempt to provide the participating 
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faculty with a sense of the reform movements and relevant pedagogical practices 
within the fields of mathematics and science. 

Attendance at the Collaborative workshops has been limited and has not 
grown over the years of funding. Although these activities provided opportunities 
for individuals to meet other participants and to hear about their work, only 
approximately 20 faculty members attended these Collaborative efforts on a 
continuous basis. A secondary potential goal of these meetings would be to 
increase the numbers of faculty involved in the Collaborative. This did not occur. 

Product specifications for NYCETP course dissemination. There were efforts 
during the first and second year to come to terms with what faculty who were paid 
to revise and/or develop courses should provide to the Collaborative. The issue 
about documentation of courses arose since institutionalization of reform is a 
concern. In the CUNY context many Liberal Arts and Science courses taken by 
future teachers are taught by adjunct instructors and rarely are there provisions for 
continuity. During the second year of the Collaborative, the evaluation team 
developed a draft, entitled "Product Specifications for NYCETP." The draft intended 
to suggest the necessary rationale and detail to help others, such as adjuncts, 
understand the nature of the revised course. 

An ad hoc committee of the Internal Advisory Board was appointed to 
consider and revise this draft. The document was reduced from two pages to one 
and included the requirement that one or two class sessions from the course be 
detailed. (As an aside, feedback related to the case study process from several 
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faculty members stated that the expectations for the course revisions and 
development should have been clear before they started work. The product 
specifications were an attempt to provide faculty members with such 
expectations.) By the middle of year four there was no implementation of the 
revised course specifications, with two exceptions. In these two instances, the 
faculty used the course specifications in providing documentation to the 
Collaborative (a math course at one college and chemistry course at another). 
Distribution and use of course documents has not occurred. 

There are, however, two main sources of data from which the evaluators 
can draw conclusions related to the fidelity of course reform. These are the peer 
review process and the case studies. In both of these evaluative activities faculty 
and evaluators directly provided feedback to the Pis and to the individuals revising 
the courses, feedback about the degree to which the courses, as documented, met 
the goals of the Collaborative. 

Peer Review Process. The peer review process was undertaken to provide 
feedback to faculty revising and developing courses for NYCETP. Five main 
categories of ratings are provided to reflect the degree to which the course 
documents provide evidence of meeting specific criteria for: (1) student centered 
instruction; (2) explicit inclusion of the present Standards documents for Science 
and Mathematics; (3) course/materials minimum expectations; (4) evidence of 
effectiveness or student outcomes; and (5) NYCETP general objectives. The degree 
to which the individual courses met each of these goals was rated on a four-point 
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scale. The lowest score (1 ) reflects major problems with the document in this area. 
The highest score (4) reflects an excellent rating with respect to this category of 
goals. Each of the larger categories consists of several components (see Appendix 
A; NYCETP Guidelines for Self-Study of Course Documents/Curriculum). 

These guidelines were used to review each course document by at least one 
faculty member or evaluator. Some courses have more than one rater. Where this 
occurred, we found inconsistencies in the ratings given. For example, there were 
four reviewers for one course. On several items (e.g., use of inquiry-based 
approach, focus on deep understanding of major concepts, and detailed content), 
two of the reviewers indicated that there was evidence that the items were present 
in the course document, and two of the raters indicated that the items were 
absent. Individual reviewers came to this process with different expectations or 
criteria about what constituted a "focus on deep understanding," for example. (As 
a result of earlier discussions, the evaluators produced a Glossary of Terms 
(Appendix B) to accompany the Peer Review Guidelines to help individual faculty 

better understand the terms used in the outline.) 

The Campus Peer Review Summary Table (Appendix C) provided us with an 
overall rating for each category. The courses reviewed varied in the degree to 
which they met each of these goal categories. Overall, the course documents 
ranged from a mean rating of 1 .75, indicating major and/or minor problems with 
the document, to a 3.45, indicating a good to excellent rating; one course received 
perfect ratings for each category. Out of the total of 13 courses reviewed, two 
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were rated unsatisfactory with mean scores across all categories of 1.75, four had 
minor problems (2.00 - 2.50), three were good (2.50 - 3.00), three courses were 
good to excellent (ratings greater than 3.00), and one was rated as excellent in all 
categories. Overall the ratings of the 1 3 course documents provide some indication 
that these courses are more student centered (e.g., more use of inquiry-based 
approaches, focus on deeper understanding, and/or an emphasis on problem 
solving and critical thinking). 

Case studies of Collaborative courses. Tittle and Pape (1996) developed a 
framework to describe and classify procedures and instruments developed to 
measure reform in school mathematics and science classrooms. This framework is 
based on the classification of teacher and student activities and interactions in 
classroom processes. It emphasizes features of reform-based classrooms such as 
subject matter, classroom interactions, types of knowledge and cognitive 
processes, methods and procedures, and teacher knowledge and beliefs. 

Drawing on this framework, a Case Study Outline was developed to provide 
faculty members with guidelines for writing the case studies. The case studies 
were developed through meetings with course faculty, campus visits and 
observations (if feasible), and examination of course documents. The case studies 
were initiated in year one, and we conducted a follow-up interview with each 
faculty member involved. These interviews lead to the revision of the outline and 
the final version (Appendix D) includes six broad areas of inquiry: (1) general 
overview and context; (2) students -- target population; (3) course revisions; (4) 
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new/revised course specifications (e.g., course syllabi, instructional activities and 
materials); (5) student outcomes and assessments; and (6) faculty roles. Year one 
case studies (1996) provided baseline data or information about the courses before 
revision as well as faculty practices and beliefs at that time. Year two and three 
(1997, 1998) case studies provided feedback to the faculty. 

Year one case studies included six mathematics courses and one science 
course distributed among the six campuses. Seven faculty members from various 
campuses studied these courses. During year two, three science courses and one 
mathematics course were case studied by four faculty. One of the courses was an 
experimental course in mathematics and another one served as an example of a 
fully-developed course that was revised prior to the Collaborative. During spring 
1998, three courses, a mathematics course, a physics course, and a technology- 
based science education course, were reviewed. One of the courses studied during 
year one as baseline data was examined again in year three when a different 
instructor taught the course. For some of these case studies there was a reciprocal 
arrangement (i.e., faculty from campus A studied a course at campus B and vice 
versa) and some faculty were involved with more than one case study. In other 
cases both a Liberal Arts and Sciences and a Teacher Education faculty member 
were involved in a single case study. 

In all, a total of 1 5 case studies have been conducted. Each case study 
involved faculty from different campuses, and, typically. Liberal Arts and Science 
faculty and Teacher Education faculty. The in-depth examination of specific 
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courses by both the individual teaching the course and the faculty member who 
wrote the case study served as both a means of staff development and, for many, 
a vehicle through which the process of course revision was begun. That is, while 
the case studies provided baseline data related to the courses to be revised, they 
also resulted in faculty visiting other campuses with a specific goal and focusing 
their interactions and discussions on particular aspects of the curricula. These 
individuals later reported conversations related to teaching activities, to the 
selection of topics included in the particular courses, and to the assessment of 
student understandings and attainments, among others. In some instances, the 
case studies served to focus the course revisions more concretely across 
campuses. 

The fidelity of course reform as evidenced by the faculty case studies. 

Review of the case studies documents yields findings similar to those for the peer 
review of Collaborative course documents. The majority of courses have made 
some progress toward meeting NCTM and NRC standards and a few are 
exemplary. 

One course studied during 1 998 was an education course. The case study 
describes the instructor's use of the major aspects of computer technology, and a 
syllabus and web links for the spring 1998 semester (URL: http//www. nyu.edu/ 
classes/murfin/micro/index.html). Included on the web site are assignments on 
developing web pages, group projects, lesson plans, research paper or grant 
proposal, along with student accomplishments. The course includes "tool software" 
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as well as other instructional software and is required for undergraduate students in 
the mathematics and science teaching programs. The course has changed from its 
pre-1996 format (i.e., evaluation of software packages) to the present emphasis on 
the WWW and the availability of mathematics and science related sites. There is 
more evidence of links with the schools for this course. For example, one activity 
was specifically designed to involve students with a master teacher to develop an 
interactive, on-line science fair. Also, some students from this course have been 
involved in teaching a summer program on technology for high school students. 

Another 1 998 case study provides perspective on courses under 
development. The Mathematics course has been under revision at one College since 
the beginning of the collaborative. Yet the case studies from 1996 and 1998, and 
the course documentation, suggest that there is still no clear direction for the 
course, and its extension, a second Mathematics course. Both courses are required 
and are key courses for prospective students in the elementary education program 
at the College. The course is taught by several faculty in the Department of 
Mathematics, with little apparent agreement. The review also indicates no 
collaboration with other NYCETP campuses in the course development, although 
there is contact with the education faculty on the campus. 

Overall, the case studies for the three years now provide several examples of 
course documentation or "products" for the Collaborative. Eliminating courses to be 
discontinued, duplicate case studies, and/or courses with unsatisfactory ratings, 
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there are a total of 8 case studies in which there is sufficient documentation to 
enable other faculty to consider adoption and/or adaptation. 

Unanticipated outcomes of the case studies include the importance of the 
interactions in fostering collaboration between faculty members on different 
campuses and in developing their understanding of reform-based teaching and 
learning. Through in-depth visitations of the various campuses, such things as the 
facilities available and/or necessary to provide reform-based courses became clear 
to the faculty. One faculty member (in a Mathematics Department) reported that 
she was better prepared to provide a request for space and materials than she had 
been prior to writing the case study. 

With respect to changes in the faculty member's thinking, many of the 
faculty reported that the case study process influenced their thinking related to the 
revision of their courses. The following were mentioned as changes due to 
participating in the case study: the incorporation of more computer graphics and 
simulations, an evaluation of the entrance requirements for particular courses, the 
need to increase collaboration among students and the use of manipulatives as 
integral parts of the course, and the need for greater coherence between math and 
math education courses. For some, the difficulty inherent in collaboratively revising 
courses (i.e., collaboration between Liberal Arts and Sciences faculty and Education 
faculty) became more apparent. These two groups of individuals typically come to 
the revision process from different perspectives. One professor wrote: 

I am experiencing first hand how difficult it is to plan and tryout activities 
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with colleagues who are not committed to course revision that requires 
major change in principles of teaching - such as deciding to cover fewer 
topics in a course but covering them in more depth, or taking sufficient time 
for hands-on activities and follow-up discussion before moving to a lecture- 
discussion of materials. At one of the meetings I heard faculty speak of 
themselves as a biologist or chemist but not as educators, and this causes 
me much concern because it suggests that we are not beginning with the 
same goals and expectations for courses and instruction. Perhaps faculty at 
different ends of the continuum on such issues will each move closer to a 
middle position, on the positive side, I am happy with students' comments 
about tryout of some activities. The question is how do our efforts lead to 
substantive change in the actual courses - especially in content areas. 

In summary, the use in evaluation of NYCETP faculty to conduct the case 
studies and to review course documents of new and revised courses served many 
of the goals of the Collaborative and focused the curriculum revisions during the 
following years of the project. The case study was reported to have served to 
strengthen one professor's commitment to the Collaborative efforts and increased 
the potential for collaboration with members on other campuses. Finally, the 
resulting discussions facilitated individual faculty member's course revisions, one of 
the main formative goals in the beginning stages of the Collaborative. 
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Critical Analysis of Evaluation Practices and Project Outcomes 

There are many specific aspects of the NYCETP outcomes to date that can 
be identified; however, our focus here is on the fidelity of reforms of courses. First, 
on the positive side of the evaluation, from the perspective of the fidelity of course 
reforms and drawing on our earlier analyses of course documents, the peer review 
process, and the case study documents, there are 10 course documents for 
courses still being taught. Seven of these are exemplary and/or well-reviewed 
courses that match many of the goals of the national standards in mathematics and 
science. These evaluation practices did help to increase the number of available 
course documents and to focus attention on what was being done (or, in many 
instances, not done). The focus on course changes and the structures of the peer 
reviews and case studies emphasized important aspects of reform as it appears in 
classrooms. As Lipsey (1993, 1997) and Weiss (1997) have argued, we located a 
key mechanism of change in the classroom — content, interaction processes, 
assessments, and so on. We viewed the attainment of reform classrooms as a pre- 
requisite to changing models of teaching and student outcomes, and, indeed, the 
original project proposal included a list of courses on each campus to be revised 
and a provisional schedule for revision. 

Second, on the critical side of the evaluation, the case studies and peer 
review processes were put forward by the evaluators as a means of helping the 
project reach its goals, that is as an implementation strategy rather than purely an 
evaluation procedure. On three of the individual campuses there were sufficient 
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local reform initiatives that course revisions were initiated and continued to expand. 
This did not necessarily mean that the courses were sufficiently documented to be 
disseminated either on their own campuses or between campuses. Nor does it 
mean that course revisions initiated were always completed, since New York State 
and CUNY campus curriculum demands on both the Arts and Science and 
Education faculty make course reforms a moving target! 

Also on the critical side, the project and the evaluation lacked focus on what 
in retrospect is an important key mechanism, as important as the classroom: 
faculty understanding and experiences related to changing classroom practice-that 
is, the faculty development process. There was continuous feedback to Pis from 
comments on the conference, workshop, and curriculum group meeting feedback 
surveys about the need for more depth in understanding of changes, of classroom 
models to view and/or observe, and so on. These comments were duly noted and 
emphasized to the Pis. The steps to take for faculty development were not clearly 
identified in the project proposal, and no direct approach to this problem was 
developed by the project. Limited approaches or steps were taken in some of the 
curriculum and working group meetings. 

The needs and challenges in faculty development can be inferred from the 
existing literature on K-12 teacher professional development (e.g., Knapp, 1997; 
Fennema & Nelson, 1998). However, information about the programmatic 
processes, experiences, and time needed for change for college faculty is not well 
documented. The NSF CETP Guidelines for Reform (1998) contain a section on 
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faculty enhancement. This section does suggest three areas of activities: 1) mini- 
grants; 2) brown bag lunches, seminars, week-long topical workshops, and 
retreats; and 3) make sure faculty are familiar with the national standards. The 
effectiveness and context for their suggestions are not described. 

There are some likely differences between pre- and post-secondary 
professional development or enhancement. Faculty members think about courses, 
not curricula, and the rewards for "excellence" in teaching are few. And, faculty in 
mathematics and science typically do not exhibit a curiosity about their students' 
growth and intellectual development and how their teaching may connect (or 
disconnect) with student learning in their own classes. Further, individual faculty 
who undertake revising their teaching and courses often are isolated in 
departments (in mathematics and science) where other faculty are resistant or 
uninterested in their teaching and its connection with student learning. 

Implications and Generalizations 

For evaluation practice. The immediate implications for evaluation 
practices from the NYCETP evaluation are in at least two areas. First, limitations of 
formative evaluation for performing the funder's role in strengthening or monitoring 
project staff, and second, recognition that the issue is more than just evaluation 
when formative/internal evaluators find themselves designing evaluation activities 
continually to fill in for project activities. 

As mentioned early in our paper, a key aspect of any project functioning is 
located in the leadership/administrative staff. After the initial meetings of the Pis, 
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the evaluator met with the project director and discussed the type of knowledge 
needed by a senior staff member to support the project director, since the project 
director was only able to devote limited time to the project. Project leadership and 
infrastructure both in terms of time commitments and the knowledge base of 
personnel is a critical area about which evaluators and funders need to be more 
realistic. Evaluators and funders need to carefully examine the staffing structure 
especially in large-scale reform projects at the post-secondary level. Based on this 
CETP case study, the project leadership/administrative structures cannot be funded 
as typical university research projects, with faculty time allocations that are part- 
time. 

Also, as mentioned earlier, the case studies and peer review processes were 
primarily evaluation responses to the need to enhance structure and direction for 
faculty who began to undertake course reforms, and the need to enhance the 
definition of faculty and project responsibility in course reforms. These evaluation 
activities attempted to augment project leadership and definitions of 
responsibilities. The "usual" procedures of independent faculty performing at their 
own pace and defining their own goals, as found in academic settings, was not a 
viable model for a project with time lines, due dates, and "product" goals. 
Continued feedback on these issues did not produce the optimum level of response 
by all involved in the project. Again, evaluators can provide technical assistance 
and consulting but cannot direct projects. The dilemma is not easily resolved. In 
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this project the evaluators had some positive contribution, but often not as much 
as was needed to bring the project to a higher standard. 

For evaluation theory. From the standpoint of evaluation theory, the 
implications are less clear. One implication may be that the reliance on theory 
developed from evaluations on education at the K-12 level has some value, but 
lacks sufficient fit to post-secondary education to be useful given the context of 
academic settings in which the CETP are embedded. A second implication arises 
from the recent writings on systemic reform and logic models, as described in 
papers from the National Institute for Science Education Fourth Annual Forum on 
Evaluation of Systemic Reform (NISE, 1999). Although these again are primarily K- 
12 focused, the projects are larger and the evaluation issues have apparent overlap 
with the CETP projects and postsecondary reforms. The scope of the systemic 
projects has led evaluators to look toward modeling of systems and change 
(Ridgeway, 1998 as cited in Heck, 1999) and to logic models (Rog, 1994 as cited 
in Barley, 1 999) as ways of integrating theory-driven evaluation with project 
activities. Bringing these ideas to the context of a six campus project and focusing 
on the project design/plan/activities and the key mechanisms of classroom and 
faculty professional development in teaching might have had some effect in 
focusing evaluation theory and practice more tightly on the project proposal, 
activities, and implementation. This might have enhanced interactions particularly 
with Pis and NSF. 
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For research on evaluation of post-secondary reform of teaching and 
curricula. From the NYCETP perspective, research is needed on the existing 
evaluations in post-secondary education and teacher preparation. The context of 
the CETPs, at least the NYCETP, indicates that the project leadership structures, 
the type and amount of faculty involvement (e.g., course release and summer 
salary), teaching rewards, and faculty role definitions may lead to different 
evaluation issues in post-secondary and K-12 evaluations. A systematic analysis of 
existing evaluations, contrasting similarities and differences in evaluation practice 
and outcomes, may be useful. 

Implications for Funding Agencies 

The implications for funding agencies are in the grant making and funding 
processes: 

1 . The requests for proposals; 

2. The review process for proposals; and 

3. The monitoring of grants. 

The request for proposals. Based on the evaluation and project 
implementation of the NYCETP, problems can be identified early in the funding 
process. Requests for Proposals (RFPs) can be developed based on a review of 
existing information about similar projects and their evaluations. Funding agencies 
should commission reviews (externally if necessary) to examine the findings from 
evaluations and projects in the area of reform of teacher preparation programs and 
to draft critical summaries. These summaries can be used to inform future 
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applicants. These reviews would identify areas such as faculty development, and 
contrast characteristics of successful and unsuccessful projects, and their 
evaluations. Characteristics would include areas identified in this evaluation. For 
example, project leadership structure, the detail of planning, the program theory 
and logic/activities links, and so on. Such information would assist applicants and 
reviewers. 

The review process for proposals. The review process should include 
projections of project likelihood of success, given the important characteristics 
identified for teacher preparation projects involving faculty. Budgets and time 
allocations should be critically judged in relation to staffing and the scope of 
proposed activities. 

The monitoring of grants. If the proposal and review processes are more 
detailed and carefully described, as with flow charts and activities, identification of 
deadlines, etc., funders will have a means to make decisions about continuing 
funding. If proposals are requested for areas in which funders have little knowledge 
and the existing literature on an area is limited, such as faculty development in 
mathematics and the sciences, then smaller, more targeted proposals should be 
considered for funding. This strategy should be aimed at developing the information 
for larger scale projects. These smaller projects would be able to carry through the 
faculty development process, the change in courses (teaching and learning 
processes), and document changes in student outcomes. When there is a base of 
models that warrant application, larger projects can be developed. 




28 



NYCETP Context and Evaluation 28 



In summary, both funders and evaluators have similar issues with the CETP. 
These involve careful review of project proposals with awareness of critical 
features that are a minimum for successful implementation, and use of early 
warning systems for detecting problems in individual projects. Evaluators and 
funders differ, however, in their final judgments. Where evaluators may serve less 
as judge and more as consultant (Century, 1999), funders ultimately have 
responsibilities to a broader constitutency and may need to serve more as judges. 
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Appendix A 

NYCETP GUIDELINES FOR SELF-STUDY OF COURSE DOCUMENTS/CURRICULUM 

The Guidelines for Self-Study have been prepared by the NYCETP. The Guidelines are intended to 
structure your review of the materials prepared for dissemination with Collaborative support. Ratings 
and WRITTEN COMMENTS are necessary. Both will serve as recommendations for dissemination or 
further revision of materials. 

Reviewer Name/College: Date: 

Course number and title: 

Faculty who revised course/College: 



Self-Study for course curriculum and materials: 

Use the Guidelines for Self-study to guide your review and ratings. Rate each course/curriculum 
package on the categories below. Your ratings need to be explained. ATTACH WRITTEN COMMENTS 
on your ratings to assist in the improvement of the materials. 

1) Rate the individual items in each category: CIRCLE ONE 1 2 3 NA 

1- Absent 2-Partially present 3- Present NA-Not Applicable 

2) After you have rated all items, make an OVERALL RATING for the whole category: CIRCLE ONE 1 
2 3 4 

1 -Major Problems 2-Minor Problems 3-Good 4-Excellent 



*NOTE: Terms followed by an asteriks are defined in the attached glassary of terms. 



Meets Collaborative Student Centered Instructional Goals -- CIRCLE to indicate USE OF ANY OF 
THE FOLLOWING: 

12 3 NA Use of inquiry-based approaches* 

12 3 NA Opportunities for hands-on, experiential learning* 

12 3 NA Focus on deep understanding of major concepts* 

12 3 NA Emphasis on problem solving, critical thinking skills* 

12 3 NA Use of collaborative learning groups* 

12 3 NA Use of technology in effective ways 

1 2 3 NA Incorporation of alternative assessment approaches* 

1 2 3 NA OTHER (describe) 



Student Centered Instructional Goals: 



OVERALL RATING 
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B. 



Meets Collaborative Course Content Goals - CIRCLE to indicate USE OF ANY OF THE 



FOLLOWING: 
1 2 3 NA 

1 2 3 NA 

1 2 3 NA 

1 2 3 NA 



National Council of Teachers of Mathematics -- Curriculum Standards 
National Research Council — National Science Education Standards 
New York State Teacher Education test guidelines (LAST) 
OTHER(describe) 



Course Content Goals: 



OVERALL RATING 12 3 4 



C. 



Meets Collaborative Course/Materials Minimum Expectations -- CIRCLE to indicate USE OF ANY 
OF THE FOLLOWING: 

1 2 3 NA Detailed content (concepts, process skills & habits of mind*), including syllabus, 

objectives, topics, exercises, details of technology, readings, assignments 

12 3 NA Details of pedagogy, including active learning instructional strategies*, desired student 

outcomes*, and traditional and/or alternative assessment techniques* to be used when 
teaching the course, typical classroom, supporting equipment, facilities 

12 3 NA Detailed course context, including location of course in teacher education 

preparation/liberal arts and science requirements target audience -"typical" students and 
intended population 



1 2 3 NA 

1 2 3 NA 

1 2 3 NA 



Descriptions of course preparation include involvement of master teachers, liberal A & S 
faculty, education faculty, field sites, city resources (formal & informal), teaching 
assistant preparation (training sessions/training manual) 

Describes relation between course content topics and teacher preparation goals* 

OTHER (describe) 



Course/Materials Minimum Expectations: OVERALL RATING 12 3 4 

D. Meets Collaborative Evidence of Effectiveness Goals -- CIRCLE to indicate evidence of each of the 
following: 

MATHEMATICS &/OR SCIENCES 



1 


2 


3 


NA 


Conceptual knowledge/Understanding concepts 


1 


2 


3 


NA 


Problem solving (e.g., emphasis on processes - Math) 


1 


2 


3 


NA 


Theorizing and analyzing* (Sci) 


1 


2 


3 


NA 


Communicating (i.e., oral &/or written) 


1 


2 


3 


NA 


Using tools (e.g., protractor, calculator, etc.) 


1 


2 


3 


NA 


Knowledge of facts and skills/routine procedures 


1 


2 


3 


NA 


Investigating natural world* (Sci) 


1 


2 


3 


NA 


Making Connections (i.e., within subject, across subject, & to every day life - Math) 


1 


2 


3 


NA 


Reasoning: Conjectures and proof (Math) 


1 


2 


3 


NA 


Habits of mind* 
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D. Meets Collaborative Evidence of Effectiveness Goals -- CIRCLE to indicate evidence of each of the 
following (continued): 

STUDENT ATTITUDES/OTHER OUTCOMES 
12 3 NA Attitudes* 

12 3 NA Increasing interest 

12 3 NA Careers 

1 2 3 NA Safety* (Sci) 

12 3 NA Participation* 



Evidence Of Effectiveness Goals: 



OVERALL RATING: 1 



E. Evidence of NYCETP Objectives : 



A 


P 


NA 


Collaboration with other NYCETP campuses 


A 


P 


NA 


Alternative assessment methods 


A 


P 


NA 


Partnership of science/math faculty and education faculty 


A 


P 


NA 


Partnership of science/math faculty and K-1 2 teachers 


A 


P 


NA 


Incorporates the use of urban context (e.g. f informal science and mathematics 
institutions) 



F. Meets Collaborative Dissemination Goals: 



A 


P 


NA 


Printed copy 


A 


P 


NA 


Computer diskette 


A 


P 


NA 


Videotape 


A 


P 


NA 


Publisher-texts 


A 


P 


NA 


Web site 


A 


P 


NA 


Professional conference 


A 


P 


NA 


Professional journal 










A 


P 


NA 


OTHER (describe) 











Dissemination Goals: 



OVERALL RATING: 12 3 4 
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Appendix B 

NYCETP GUIDELINES FOR SELF-STUDY OF COURSE DOCUMENTS/CURRICULUM 

GLOSSARY OF TERMS 

Source: National Science Education Standards (National Research Council, 1996) 

1. Inquiry-based Approaches : "Scientific inquiry refers to diverse ways in which 
scientists study the natural world and propose explanations based on the evidence 
derived from their work." 

Inquiry: "... multifaceted activity that involves making observations; posing 
questions; examining books and other sources of information to see what is already 
know; planning investigations; reviewing what is already known in light of experimental 
evidence; using tools to gather, analyze, and interpret data; proposing answers, 
explanations, and predictions; and communicating the results." 

"Emphasizing active science learning means shifting emphasis away from teachers 
presenting information and covering science topics. The perceived need to include all 
the topics, vocabulary, and information in textbooks is in direct conflict with the central 
goal of having students learn scientific knowledge with understanding." (p. 21) 

2. Experiential Learning : "Use appropriate tools and techniques to gather , analyze, and 
interpret data. ... The use of tools and techniques, including mathematics, will be guided 
by the question asked and the investigations students design." 

Abilities necessary to do scientific inquiry: 

- "Identify questions that can be answered through scientific investigation." 

- "Design and conduct a scientific investigation." 

- "Use appropriate tools and techniques to gather, analyze, and interpret data." 

- "Develop descriptions, explanations, predictions, and models using evidence." 

- "Think critically and logically to make the relationships between evidence and 
explanations" (p. 145). 

3. Deep Understanding of Major Concepts : Students should develop "... productive 
and insightful ways of thinking about and integrating a range of basic ideas that explain 
the natural and designed world." 

"As a result of activities in grades K-12, all students should develop understanding 
and abilities aligned with the following concepts and processes: 
systems, order, and organization 
evidence, models, and explanation 
constancy, change, and measurement 
evolution and equilibrium 
form and function" (pp. 115-116). 
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4. Critical Thinking : "Thinking critically and logically to make the relationship between 
evidence and explanations." 

"Thinking critically about evidence includes deciding what evidence should be used 
and accounting for anomalous data. Specifically, students should be able to review data 
from a simple experiment, summarize the data, and form a logical argument about the 
cause-and-effect relationships in the experiment." (p. 145) 

5. Collaborative Learning Groups : Students should ". . . engage in problem solving, 
planning, decision making, and group discussions." (p. 20) 

- "Using a collaborative group structure, teachers encourage interdependency among 
group members, assisting student to work together in small groups so that all participate 
in sharing data and in developing group reports." 

- The teacher's role in these small and larger group interactions is to listen, 
encourage broad participation, and judge how to guide discussion — determining ideas to 
follow, ideas to question, information to provide, and connections to make" (p. 36). 

6. Alternative Assessment Approaches : "Rather than simply checking whether 
students have memorized certain items of information, new assessments probe for 
students understanding, reasoning, and use of that knowledge — the skills that are 
developed through inquiry." 

- "Besides conventional paper and pencil tests, assessments might include 
performances, portfolios, interviews, investigative reports, or written essays" (p. 6). 

- "Teachers of science engage in ongoing assessment of their teaching and of 
student learning. In doing this, teachers use multiple methods and systematically gather 
data about student understanding and ability." (p. 37) 

7. Habits of Mind : (TIMSS) - "curriculum encourages ways of scientific and 
mathmatical thinking such as openness, objectivity, tolerance of uncertainty, 
inventiveness, curiosity" (Robitaille et al., 1993, p. 84). 

8. Active Learning Instructional Strategies : "Inquiry into authentic questions generated 
from student experiences is the central strategy for teaching science." 

- "Teachers focus predominantly on real phenomena, in classrooms, outdoors, or in 
laboratory settings, where students are given investigations or guided toward fashioning 
investigations that are demanding but within their capabilities." 

- "... teachers can take an inquiry approach as they guide students in acquiring 
and interpreting information from sources such as libraries, government documents, and 
computer databases -- or as they gather information from experts from industry, the 
community, and government." 

- "Teachers of science guide and facilitate learning. In doing this, teachers 

- Focus and support inquiries while interacting with students. 

- Orchestrate discourse among students about scientific ideas. 

- Challenge students to accept and share responsibility for their own learning. 
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- Recognize and respond to student diversity and encourage all students to 
participate fully in science learning. 

- Encourage and model the skills of scientific inquiry, as well as the curiosity, 
openness to new ideas and data, and skepticism that characterize science." (p. 32) 

9. Desired Student Outcomes : " What students learn is greatly influenced by how they 
are taught." 

- "The actions of teachers are deeply influenced by their perceptions of science as 
an enterprise and as a subject to be taught and learned." 

- "Student understanding is actively constructed through individual and social 
processes." 

- "Actions of teachers are deeply influenced by their understanding of and 
relationships with students." (p. 28-9) 

- "Select science content and adapt and design curricula to meet the interests, 
knowledge, understanding, abilities, an experiences of students." 

- "Select teaching and assessment strategies that support the development of 
student understanding and nurture a community of science learners" (p. 30). 

10. Course Content Topics and Teacher Preparation Goals : "Program Standard A: All 
elements of the K-12 science program must be consistent with the other National 
Science Education Standards and with one another and developed within and across 
grade levels to meet a clearly stated set of goals." 

- "In an effective science program, a set of clear goals and expectations for students 
must be used to guide the design, implementation, and assessment of all elements of the 
science program." 

- "Teaching practices need to be consistent with the goals and curriculum 
frameworks." 

- "Assessment policies and practices should be aligned with the goals, student 
expectations, and curriculum frameworks" (p. 21 1). 

1 1 . Theorizing, Analyzing, Investigate : see the following: Inquiry, deep understanding 
of major concepts, critical thinking, experiential learning, active learning instructional 
strategies 

12. Investigating Natural World : see the following: Inquiry, deep understanding of 
major concepts, critical thinking, experiential learning, active learning instructional 
strategies 

13. Attitudes : "Teachers of science develop communities of science learners that reflect 
the intellectual rigor of scientific inquiry and the attitudes and social values conducive to 
science learning. In doing this, teachers 

- Display and demand respect for the diverse ideas, skills, and experiences of all 
students. 
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- Enable students to have a significant voice in decisions about the content and 
context of their work and require students to take responsibility for the learning of all 
members of the community. 

- Nurture collaboration among students. 

- Structure and facilitate ongoing formal and informal discussion based on a shared 
understanding of rules of scientific discourse. 

- Model and emphasize the skills, attitudes, and value of scientific inquiry." (pp. 45- 

6 ) 

- "Teachers who are enthusiastic, interested, and who speak of the power and 
beauty of scientific understanding instill in their students some of those same attitudes." 
(p. 37) 

14. Safety : "Ensure a safe working environment." 

- "Teachers also teach students how to engage safely in investigations inside and 
outside the classroom." (p. 44) 

15. Participation : "Challenge students to accept and share responsibility for their own 
learning." 

- "Although open exploration is useful for students when they encounter new 
materials and phenomena, teachers need to intervene to focus and challenge the 
students, or the exploration might not lead to understanding." 

- "Premature intervention deprives students of the opportunity to confront problems 
and find solutions, but intervention that occurs too late risks student frustration. 

Teachers must also decide when to challenge students to make sense of their 
experiences: at these points, students should be asked to explain, clarify, and critically 
examine and assess their work." (p. 36) 

- "Orchestrate discourse among students about scientific ideas." 

- "Recognize and respond to student diversity and encourage all students to 
participate fully in science learning." 

- "Teachers monitor the participation of all students, carefully determining ... if all 
members of a collaborative group are working with materials or if one student is making 
all the decisions. This monitoring can be particularly important in classes of diverse 
students, where social issues of status and authority can be a factor." (pp. 36-7) 
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Appendix D 

NYCETP Year 3 Case Study Outline 

Begin the case study with an introductory page that describes the special characteristics of 
this course relative to the NYCETP collaborative goals: 

• Increased use of inquiry-based approaches. 

• Opportunities for hands-on, experiential learning. 

• Focus on deep understanding of major concepts. 

• Increased use of technology in effective ways. 

• Use of an "urban context." 

• Incorporation of alternative assessment approaches. 

• Partnership of science/mathematics faculty and education faculty. 

• Partnership of science/mathematics faculty and K-12 teachers. 

Use the following outline to guide your documentation of the Collaborative course. The goal 
of providing this outline is to attain similarity between the case studies on different 
campuses. Please use the categories and questions to guide your interviewing and writing. 
Collect alj materials that are available for the course and that may be included to illustrate 
your case descriptions. 

I. General Overview and Context: 



1. Instructor's name, department, title/position, course relevant 
experience/education, years teaching this course. 

2. Title of course, number of hours and credits. 

3. Catalogue description, placement of course in sequence, required core courses or 
other prerequisites, required course or elective. 

4. Typical number of sections offered per semester, description of faculty who 
teach sections other than interviewee (i.e., number and percentage full-time 
versus adjunct or part-time faculty, pertinent experience, etc.). 

5. Entering requirements such as grade point average or score on a diagnostic 
assessment. 

6. Organization or breakdown of hours for each class session or hours per week 
(i.e., lecture, recitation, and lab hours). 

7. Brief description of institution, total enrollment size, characteristics of student 
body. 
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8. Brief history of course, evolution of course over time (prior to NYCETP revisions) 
and broad goals or circumstances for changes. 

Students - Target Population: 

1. Course enrollment size, intended population (level of students, i.e., first year, 
major, teacher education student, etc.), 

2. Description of several "typical" students (i.e., age, gender, ethnicity, SES, 
background, likely career goals or major). 

Course Revisions or Development as Part of NYCETP Activities: 



1 . Name(s) of college faculty who revised/developed the course as part of NYCETP 
efforts. Others who have helped planning or revising course(s)? Did members of 
the Liberal Arts faculty and the Education faculty collaborate on the revision or 
development of the course (describe collaborative efforts)? 

2. Overview of new or revised course including characteristics specifically related to 
particular NYCETP goals. 

3. When was the new or revised course offered for the first time (or when will it be 
offered)? Will it be offered again? When? 

4. Differences between new/revised course and the course that was originally 
offered (e.g., how has course structure or allocation of class time changed, how 
are goals and expected outcomes different, etc.). 

5. Does this course involve the collaboration of experienced or master teachers, 
school district coordinators or others? Are exemplary field sites (i.e., classroom 
observation or student-teaching sites) being developed in conjunction with the 
new or revised course? Describe the use of such sites. 

6. How is the impact of revisions on prospective teachers being evaluated? Have 
additional revisions been planned as a result of such an evaluation of the revised 
course? 

7. Plans for revision not yet implemented or fully developed. Do these plans include 
provisions for the recruitment of teachers? 

8. How has revising or developing the course changed your (i.e., the faculty who 
revised the course) thinking related to learning and teaching? 
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